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Program Description 2 The Cognitive Tutor® Algebra 1 curriculum, published by Carn- 
egie Learning, is an approach that combines algebra textbooks 
with interactive software. The software is developed around 
an artificial intelligence model that identifies strengths and 


weaknesses in each individual student’s mastery of mathemati- 
cal concepts. It then customizes prompts to focus on areas 
where the student is struggling and sends the student to new 
problems that address those specific concepts. 


Research 3 One study of Cognitive Tutor® Algebra 1 meets What Works Clear- 
inghouse (WWC) evidence standards. The study included 255 
ninth-grade students from three junior high schools in Oklahoma. 4 


Based on this study, the WWC considers the extent of 
evidence for Cognitive Tutor® Algebra 1 to be small for math 
achievement. 



Effectiveness Cognitive Tutor® Algebra I was found to have potentially positive effects on math achievement. 



Math achievement 

Rating of effectiveness Potentially positive effects 

Improvement index 5 Average: +15 percentile points 



1. This report has been updated to include reviews of 11 studies that have been released since 2005. A complete list and disposition of all studies reviewed 
is provided in the references. Additionally, one study that met standards with reservations in the previous version (Shneyderman, 2001) will now be 
eligible for review as part of the WWC high school math area. (The protocol for the middle school math area was revised to narrow the scope from 
examining any students in grades 6 to 9 to examining only those students who are attending middle schools or junior high schools. Studies examining 
students in grade 9 who are attending high school are included in the high school math area.) 

2. The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www.carnegielearning.com, 
downloaded September 2008). The WWC requests developers to review the program description sections for accuracy from their perspective. Further 
verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 1.0 (see the WWC Standards). 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

5. This number shows the average student-level improvement index for all findings across the study. 
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Absence of conflict 
of interest 



The studies Dynarski et. al. (2007) and Campuzano et. al. (2009), 
cited in the references section below, were prepared, in whole 
or in part, by staff of Mathematica Policy Research, Inc. (MPR). 



However, because the Cognitive Tutor ® Algebra I study samples 
fell outside the scope of the review, the studies were not eligible 
for review under the Middle School Math topic area. 



Additional program 
information 



Developer and contact 

Cognitive Tutor® Algebra I was developed by and is distributed by 
Carnegie Learning, Inc. Address: Frick Building, 20th Floor, 437 Grant 
Street, Pittsburgh, PA 15219. Email: info@carnegielearning.com. 
Web: http://www.carnegielearning.com/software_features.cfm. 
Telephone: (888) 851-7094. 

Scope of use 

Pilot implementation of the curriculum began in 1992 with 84 
students in one school. As of August 2008, Cognitive Tutor ® 
curricula, which include Bridge to Algebra, Algebra I, Algebra II, 
Geometry, and Integrated Math, have been used by more than 
500,000 students in approximately 2,600 urban, rural, and sub- 
urban school districts across the United States. The number of 
students solely using Cognitive Tutor ® Algebra I is not available. 

Teaching 

Cognitive Tutor ® Algebra I addresses both mathematical content 
and process standards. Generally, three periods a week are 



spent using the Cognitive Tutor® Algebra I text for classroom 
activities, and two periods are spent in the computer lab using 
the Cognitive Tutor® Algebra I software. The textbook aims to 
foster a collaborative classroom environment where students 
develop skills to work cooperatively to solve problems, par- 
ticipate in investigations, and propose and compare solutions. 
Students learn with the adaptive software at their own pace. The 
math problems are designed to emphasize connections between 
verbal, numeric, graphic, and algebraic representations. 

Cost 

Curricula can be purchased as a full license for software and text 
or as a software-only license. Cognitive Tutor® Algebra I as a 
blended model is available as a set of software and textbooks for 
a price of approximately $67.70 per student; a teacher text set of 
materials costs $85. Volume and term discounts are available, as 
well as site license models for schools purchasing the software 
only; contact the publisher for a price quote. 



Research Fourteen studies reviewed by the WWC investigated the effects 
of Cognitive Tutor® Algebra I. One study (Ritter, Kulikowich, Lei, 
McGuire, & Morgan, 2007) is a randomized controlled trial that 
meets WWC evidence standards. The remaining 13 studies do 
not meet either WWC evidence standards or eligibility screens. 

Meets evidence standards 

Ritter, Kulikowich, Lei, McGuire, & Morgan (2007) randomly 
assigned algebra course sections to the intervention or control 
curriculum to assess the impact of Cognitive Tutor® Algebra I on 



the math achievement of ninth-grade students in three suburban 
junior high schools in Oklahoma. During the 2000-01 school 
year, ten Cognitive Tutor® Algebra I classrooms were compared 
with nine classrooms using McDougal-Littell’s Heath Algebra I, a 
traditional, teacher-directed curriculum. The analysis sample for 
the end-of-course algebra assessment included ten Cognitive 
Tutor® Algebra I classrooms (153 students) and six traditional 
classrooms (102 students). 6 Each of six study teachers taught 
both Cognitive Tutor® Algebra I and traditional classrooms. 



6. In order to save on the cost of the end-of-course algebra assessment, the study authors randomly selected one control classroom for each teacher 
to take the exam. 
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Research (continued) Meets evidence standards with reservations 

No studies meet evidence standards with reservations. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 



Standards Handbook, Appendix G). The extent of evidence takes 
into account the number of studies and the total sample size 
across the studies that meet WWC evidence standards with or 
without reservations. 7 

The WWC considers the extent of evidence for Cognitive 
Tutor® Algebra I to be small for math achievement. 



Effectiveness 



Findings 

The WWC review of interventions for Middle School Math 
addresses student outcomes in the math achievement domain. 
The findings below present the authors’ estimates and WWC- 
calculated estimates of the size and the statistical significance of 
the effects of Cognitive Tutor® Algebra I on students. 8 

Math achievement 

Ritter, Kulikowich, Lei, McGuire, & Morgan (2007) reported a 
positive but not statistically significant effect of Cognitive Tutor® 
Algebra I on the Educational Testing Service (ETS) Algebra 
End-of-Course Assessment. The effect size was large enough 
to be considered substantively important according to WWC 
standards (that is, at least 0.25). 



In sum, one study showed a substantively important positive 
effect in the math achievement domain. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 



The WWC found Cognitive 
Tutoi® Algebra I to have 
potentially positive effects 
for math achievement 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Proce- 
dures and Standards Handbook, Appendix F). The improvement 
index represents the difference between the percentile rank of 
the average student in the intervention condition and the per- 
centile rank of the average student in the comparison condition. 



Unlike the rating of effectiveness, the improvement index is 
entirely based on the size of the effect, regardless of the statisti- 
cal significance of the effect, the study design, or the analysis. 
The improvement index can take on values between -50 and 
+50, with positive numbers denoting favorable results for the 
intervention group. 

The average improvement index for math achievement is +15 
percentile points for the one study. 



7. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types of 
settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating 

was determined for Cognitive Tutor ® Algebra I is in Appendix A6. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms 
or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical 
significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for 
multiple comparisons. In the case of Ritter, Kulikowich, Lei, McGuire, & Morgan (2007), no corrections for clustering or multiple comparisons were needed. 
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The WWC found Cognitive 
Tutoi® Algebra I to have 
potentially positive effects 
for math achievement 

(continued) 
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Appendix 



Appendix A1 Study Characteristics: Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007 (randomized controlled trial) 



Characteristic 


Description 


Study citation 


Ritter, S., Kulikowich, J., Lei, P., McGuire, C., & Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor ® Algebra 1. In T. Hirashima, 
H. U. Hoppe, & S. Shwu-Ching Young (Eds.), Supporting learning flow through integrative technologies (pp. 13-20). Netherlands: I0S Press. 

Additional source: Morgan, P, & Ritter, S. (2002). An experimental study of the effects of Cognitive Tutor® Algebra 1 on student knowledge and attitude. Retrieved 
November 22, 2006, from http://www.carnegielearning.com/research/research_reports/morgan_ritter_2002.pdf. 


Participants 


Participants included 426 ninth-grade students (206 treatment, 220 control) who were assigned to one of six algebra teachers in three study schools. 1 Algebra course 
sections for each teacher were randomly assigned to a curriculum. The study authors eliminated from the analysis 83 students who transferred within the district to a different 
section of the course, did not enroll in the district for the second semester, did not receive a grade, or whose records indicated a conflict between the curriculum and class 
assignment. 2 In order to reduce the cost of the Algebra 1 assessment, only one control class was randomly selected for each teacher involved in the study. The algebra assess- 
ment analysis sample included 255 students (153 intervention, 102 control) from 16 classrooms (10 intervention, 6 control). The analysis sample for the grades analyses 
included 343 students (173 intervention, 170 control) in 19 sections (10 intervention, 9 control); however, grades are a subjective measure and were not included in the 
effectiveness rating. 


Setting 


Participating students were from three junior high schools in the Moore Independent School District in Oklahoma. Moore is a suburban school district located near 
Oklahoma City. 


Intervention 


Students spent three class periods per week in group activities and classroom discussions using the Cognitive Tutor ® Algebra 1 text and two class periods working 
on problem-solving skills with the Cognitive Tutor ® Algebra / software. The intervention occurred during the 2000-01 school year, the first year of implementation 
of Cognitive Tutor® Algebra 1 for the six study teachers. 


Comparison 


Students in the control group were taught using Heath Algebra 1, a traditional textbook published by McDougal— Littell. Study authors do not provide further information 
on this curriculum. The six study teachers taught both intervention and control classrooms in each of the three schools. At the start of the study, teachers had several 
years of experience teaching Heath Algebra 1. 


Primary outcomes 
and measurement 


The study used the Algebra End-of-Course Assessment, developed by the Education Testing Service (ETS). The other two outcomes, which were not taken into account 
in the effectiveness rating, were first semester grades and second semester (final) grades. For a more detailed description of these outcome measures, see Appendix A2. 


Staff/teacher training 


All teachers implemented Cognitive Tutor® Algebra / for the first time. During the summer prior to the start of the intervention, teachers attended a four-day training course 
to familiarize themselves with the Cognitive Tutor® software and to learn teaching techniques. 



1 . The study authors excluded from the analysis two schools that did not randomly assign classrooms to a curriculum. One school did not have sufficient computer resources to implement 
Cognitive Tutor ® Algebra I. Due to a scheduling error, teachers at the other school taught either Cognitive Tutor ® Algebra I or the traditional curriculum but not both. Only the three schools that 
implemented the within-teacher random assignment design were analyzed by the study authors and included in this report. 

2. Eleven students whose records indicated a conflict between the curriculum and class assignment were excluded by the study authors due to uncertainty about their classroom experience. The 
school registrar reported that these students were assigned to the control group that received the traditional curriculum but were actually enrolled in a Cognitive Tutor ® Algebra I classroom. 
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Appendix A2 Outcome measures for the math achievement domain 



Outcome measure 


Description 


ETS Algebra End-of- 
Course Assessment 


The ETS Algebra End-of-Course Assessment included 25 multiple-choice and 15 constructed-response items, with each type of question accounting for 50% of the 
student's score. The questions were designed to assess students’ understanding of algebraic concepts, processes, and skills (as cited in Ritter, Kulikowich, Lei, McGuire, 
& Morgan, 2007). 


Math achievement grades 


First semester and second semester (final) grades were included as an additional measure of performance. Grades are a subjective measure and were not considered 
in the effectiveness rating; rather, these outcomes are presented as additional findings in Appendix A4. 
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Appendix A3 Summary of study findings included in the rating for the math achievement domain 1 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


Cognitive Tutor® Comparison 

group 3 group 4 


Mean difference 5 
(Cognitive Tutor® 
- comparison) 


Statistical 

Effect significance 7 

size 6 (at a = 0.05) 


Improvement 

index 8 



Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007 (randomized controlled trial) 9 



ETS Algebra End-of-Course Grade 9 16/255 17.41 

Assessment (5.82) 


15.28 

(5.33) 


2.13 


0.38 


ns 


+15 


Domain average for math achievement (Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007) 10 






0.38 


ns 


+15 



ns = not statistically significant 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. Additional findings from the Ritter, Kulikowich, Lei, 
McGuire, & Morgan (2007) study are not included in these ratings, but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The Cognitive Tutor ® value is the unadjusted control group mean plus the program coefficient from the hierarchical linear modeling (HLM) analysis. The standard deviation was obtained from the 
study authors. 

4. The control group mean is unadjusted. The mean and standard deviation were obtained from the study authors. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Ritter, Kulikowich, Lei, McGuire, & Morgan 
(2007), no corrections for multiple comparisons or clustering were needed because only one outcome was considered for the effectiveness rating, and the authors accounted for clustering in 
their HLM analysis. 

10. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A4 Summary of additional findings for the math achievement domain 1 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


Cognitive Tutor® Comparison 

group 3 group 4 


Mean difference 5 
(Cognitive Tutor® 
- comparison) 


Statistical 

Effect significance 7 

size 6 (at a = 0.05) 


Improvement 

index 8 



Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007 (randomized controlled trial) 9 



First semester grades 


Grade 9 


19/343 


3.22 


2.77 


0.45 


0.42 


Statistically 


+16 








(1.00) 


(1.16) 






significant 




Second semester (final) grades 


Grade 9 


19/343 


2.82 

(1.12) 


2.39 

(1.29) 


0.43 


0.38 


Statistically 

significant 


+14 



1. This appendix presents additional findings for measures that fall in the math achievement domain. ETS scale scores were used for rating purposes and are presented in Appendix A3. The 
protocol for the middle school math area states that subjective measures, such as student grades assigned by teachers, do not qualify as relevant outcome measures. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. The Cognitive Tutor ® values are the unadjusted control scores group means plus the program coefficients from the hierarchical linear modeling (HLM) analysis. The standard deviations were 
obtained from the study authors. 

4. The control group means are unadjusted. The means and standard deviations were obtained from the study authors. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Ritter, Kulikowich, Lei, McGuire, & Morgan 
(2007), a correction for multiple comparisons was needed, so the significance levels may differ from those reported in the original study; no correction for clustering was needed because the 
authors accounted for clustering in their HLM analysis. 
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Appendix A5 Cognitive Tutor® Algebra I rating for the math achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the outcome domain of math achievement, the WWC rated Cognitive Tutor® Algebra I as potentially positive. The remaining ratings (mixed, no discernible effects, 
potentially negative, and negative) were not considered, as Cognitive Tutor® Algebra I was assigned the highest applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study of Cognitive Tutor® Algebra I showed a substantively important positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important positive effects. 

Met. No studies of Cognitive Tutor® Algebra I showed a statistically significant or substantively important negative effect, and no studies 
showed indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies of Cognitive Tutor® Algebra I showed statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies of Cognitive Tutor® Algebra I showed statistically significant or substantively important negative effects. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative 
effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence 1 


Math achievement 


1 


3 




255 


Small 



1. A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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